Touch screen interface with feedback

ABSTRACT

The invention relates to a method and an electric device comprising a screen, cooperating with said screen a sensor for detecting a presence of an object on or close to said screen, a controller and a feedback element. The controller is configured to interpret said presence of the object and calculate a force value and control said feedback element to provide a feedback with respect to the force value.

TECHNICAL FIELD

Implementations described herein relate generally to user inputprocessing, and more particularly, to devices that may provide afeedback to the user upon detection of an object in vicinity and executeoperations based on the detection.

BACKGROUND

Many multimedia devices employ touch screen technology. For example, auser may be able to interact with an image on a display screen byplacing finger(s) or a pointing device (stylus) on a screen and enter acommand by moving or tapping the finger/pointing device, to achieve thedesired effect. The use of touch screen technology allows a user toeasily navigate through menus and manipulate displayed data. Touchscreen technology is also employed in various gaming applications.

The new touch screens also allow detection of the pointing device invicinity of the screen plane without an actual physical touch. Thisallows a three-dimensional interaction with the device.

When touching the screen, the user may receive a feedback in way of asound and/or visual signal or a physical (haptic) feedback.

Haptic technology is becoming more prevalent at least partially due tothe evolution of touch screen devices. Haptic feedback is the use offorces, such as vibrations, to provide a user with the sensation of“feeling” changes on a touch screen or the like. The type of vibrationsmay be controlled, e.g., slow or fast vibrations, to simulate touchingphysical objects and elements on the screen. Thus, haptic feedback mayprovide a way of addressing the inability of a user to, for example,feel the visual keys of a keypad on a touch screen display.

Several haptic technologies are available now, including but not limitedto vibration motor actuation, piezoelectric actuation, andelectro-active polymer actuation.

When implementing haptic feedback, the force of the touch is notaccounted for.

SUMMARY

Thus, at least one object of the present invention remedies the abovementioned problem with respect to current systems and provide one ormore advantages described below.

According to example embodiments, a method of providing a userinteraction is described, comprising: receiving a user input through atouch screen interface of a touch sensitive screen; calculating a forcevalue corresponding to said user input; and providing a feedback bymeans of feedback element with respect on the calculated force value.According to one embodiment the force value affects said feedback byincreasing and/or decreasing said feedback. In one embodiment thecalculating the force value further comprises: comparing an appliedforce by the user to a threshold force value; determining a differencepercentage of the applied force and the threshold force value; andcontrolling a feedback element based on a difference percentage. In oneembodiment the calculating the force value further comprises: detectinga distance between the touch screen to a pointing object, recalculatingthe distance to a force value; and controlling the feedback elementbased on a distance.

The invention also relates to an electric device comprising a screen,cooperating with said screen, a sensor for detecting a presence of anobject on or close to said screen, a controller and a feedback element.The controller is configured to interpret said presence of the objectand calculate a force value and control said feedback element to providea feedback with respect to the force value. The device may furthercomprise a signal input to said controller from said feedback element.The device may further comprise a low pass filter connected between saidcontroller and a touch screen input. In one embodiment the device mayfurther comprise a band-pass or high-pass filter connected between saidcontroller and said feedback element. According to one embodiment thedevice of comprises a force sensitive touchscreen. According to anotherembodiment, the device comprises a touch sensitive screen comprisingmeans for generating an electrical signal indicating a positionalproperty of the pointing object and an extent property of a mechanicalinteraction within a sensing zone. The device may also comprise a sensorfor capacitive and electric field sensing based on transmitting a signalby means of one or several electrodes and receiving a response withanother electrode(s). In one embodiment, the controller is configured todefine the force value on a surface of the screen as a distributed forceon an area and thereby determining relation of the screen surfaceagainst a surface and adapting the feedback.

The invention also relates to a multimedia device for displaying images,comprising a device as described above.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIG. 1 is an illustrative example employing a first example embodiment;

FIG. 2 is an illustrative example employing a second example embodiment;

FIG. 3 is an illustrative example employing a third example embodiment;

FIG. 4 illustrates schematically a cross sectional view through atouchscreen for measuring force according to one embodiment;

FIG. 5 illustrates schematically a perspectival view of a touchscreenfor measuring force according to a second embodiment;

FIG. 6 illustrates schematically a cross sectional view through atouchscreen for detecting presence of a nearby object, according to oneembodiment; and

FIG. 7 illustrates schematically steps according to one exemplary methodof the invention.

DETAILED DESCRIPTION

In the following description, for purposes of explanation and notlimitation, specific details are set forth, such as particularcomponents, elements, techniques, etc. in order to provide a thoroughunderstanding of the example embodiments presented herein. However, itwill be apparent to one skilled in the art that the example embodimentsmay be practiced in other manners that depart from these specificdetails. In other instances, detailed descriptions of well-known methodsand elements are omitted so as not to obscure the description of theexample embodiments.

Example embodiments presented herein are directed towards interactionwith a device having a touch screen utilizing a user input andapplication specific data. For example, embodiments presented herein maybe utilized in gaming applications where a user may enter commands andreceive a feedback from the device. Based on a measurement of a forcecomponent of the user input, a determination may be made as to withwhich magnitude a user may receive a feedback.

It should be appreciated that example embodiments presented herein maybe employed in a variety of applications, for example, applicationsinvolving image, movie, gaming, menu systems, or any other type ofapplications known in the art which involve a form of display.

To summarize, as illustrated in the flow diagram of FIG. 7, a force or adistance corresponding to a force on a display by means of a pointingobject of the device is detected (1). A value corresponding to force iscalculated (2). In one embodiment the calculated value may be compared(3) to a threshold value. A feedback is provided to the user by means ofa feedback element. If the force value changes (4), e.g. increases, thefeedback may be increased (5), or no change in the force (stationary)the feedback is not changed (6), or if the force value decreases, thefeedback may be decreased (7).

FIG. 1 provides a cut through an illustrative and very simplifiedexample of a first embodiment presented. An electric device, such as amobile terminal 100, comprises a touch/force sensitive display 110. Thedisplay 110 is connected to an I/O controller 111 and a controller 120.The I/O controller 111 may include a display driver 112 and user inputunit 113. The controller may further be connected to an arrangement 130for generating a feedback. A memory unit 140 may be connected to thecontroller storing instructions for the controller 120 and data forsensed force and corresponding haptic feedback control.

The feedback arrangement 130 may be an actuator generating vibrationand/or transient, such as a rotating, or linear motor, piezoelectricactuator and electro-active polymer actuator, a sound generator etc.

According to the first aspect of the invention, the idea is to sense theforce, e.g. applied on the surface of the display and the position ofthe pointing device and use the information to control the (haptic)feedback system. The term “force” as employed herein, may refer to adirect physical force applied by a pointing device on the surface or arealization of force corresponding to a distance between the surface ofthe screen and the pointing device. In the latter, a shorter distancemay be assumed as a certain amount of force, and increasing distance anincrease of amount of the force and decreasing distance a decreasingamount of force.

The term “haptic feedback” as used herein, generally refers use of thesense of touch in a user interface design to provide information to theuser, e.g. in form of vibrations from the device's vibration alarm todenote that a touchscreen button has been pressed. However, it may alsorefer to audible or visual output.

According to a second aspect of the invention the signal from theactuator is used in a closed loop control system to optimize the drivingof the actuator to substantially optimize the performance of the hapticfeedback.

FIG. 2 is a schematic of a second embodiment of the invention. Theelectric device, such as mobile terminal 100, in same way as embedmentof FIG. 1, comprises a touch sensitive display 110. The display 110 isconnected to an I/O controller 111 and a controller 120. The I/Ocontroller 111 includes a display driver 112 and user input unit 113.The controller is further connected to a feedback generating unit 130.An output signal of the feedback generating unit 130 is connected to thecontroller 120 for providing a signal feedback. Increased load, changesthe resonant frequency of the system, the driver system changes theamplitude and/or frequency of the feedback element.

The haptic feedback may also be used as an alert signal (audio and/ormechanical (vibration)).

In one embodiment, the force on the surface of the touch sensitivescreen may be defined as a distributed force on a larger area than asmall spot, which is the case when pointing with a finger or stylus.This may be used to determine whether the device is against a surface,e.g. in a pocket or case, or not and adapt the haptic feedback, e.g. foralerting. This may be used in case of, for example a cell phone in thepocket or bag of a user.

In one embodiment, the magnitude of the force on the surface of thetouch sensitive screen may affect the feedback. A harder pressure on thescreen by the user (finger), i.e. increased load, changes the resonantfrequency of the system and the driver system changes the amplitudeand/or frequency of the feedback element. The change of the feedback maybe proportional to the force applied on the screen. One example is whenscrolling a content on the screen, the harder the user presses hisfinger, the faster the content may be scrolled. Proportional to this thehaptic feedback may be varied with respect to the pressure of thefinger.

Normally, the force sensed from the user and the vibration or transientfrom the (feedback) actuator may have different frequency spectrum. Thesignal indicating the force may vary from static DC to some hertz, e.g.approx. 10 Hz and the actuator signal may range between some hertz andsome hundred hertz. These signals must be separated.

In one embodiment, as illustrated in FIG. 3, a low-pass filter 114 orsimilar is used to filter the signal from the force sensor to extractthe force applied by the user. The effect of the feedback element 140can be distinguished and monitored using, e.g. a filter, such as aband-pass or high-pass filter 115, selective at the vibration ortransient frequency generated by the element 140.

The touch sensitive screen as mentioned earlier may be configured as anysuitable human-computer interface or touch/contact surface assembly. Thetouch screen device may be any touch screen, touch pad, touch sensitivestructure, computer monitor, laptop display device, workbook displaydevice, kiosk screen, portable electronic device screen, or othersuitable touch sensitive device. The touch screen device may beconfigured for physical interaction with a user-controlled device, suchas a stylus, finger, etc. In some embodiments, the touch screen devicemay include at least one output device and at least one input device.

The touch sensitive screen device may provide haptic feedback to atleast a portion of the electronic device, which can be conveyed to auser in contact with the electronic device. Particularly, the touchscreen device can provide haptic feedback to the touch screen itself toimpose a haptic effect when the user in is contact with the screen. Thehaptic effects can be used to enhance the user experience, andparticularly can provide a confirmation to the user that the user hasapplied a certain amount of force to or interacted with the screen to bedetected by the touch screen device.

The electronic device may be any device, such as a mobile terminal, suchas a mobile radio telephone, a desk top computer, laptop computer,electronic workbook, electronic handheld device (such as a gamingdevice, personal digital assistant (“PDA”), portable e-mail device,portable Internet access device, calculator, etc.), kiosk (such as anautomated teller machine, ticking purchasing machine, etc., printer,point-of-sale device, game controller, or other electronic device.

The controller may be a general-purpose or specific-purpose processor ormicrocontroller for managing or controlling the operations and functionsof the electronic device. For example, the controller may bespecifically designed as an application-specific integrated circuit(“ASIC”) to control output signals to a driver of the input/outputdevices to provide haptic effects. The controller may be configured todecide, based on predefined factors, what haptic effects are to beprovided, the order in which the haptic effects are provided, and themagnitude, frequency, duration, and/or other parameters of the hapticeffects. The controller can also be configured to provide streamingmotor commands that can be used to drive the haptic actuators forproviding a particular haptic effect. In some embodiments, thecontroller may actually include a plurality of processors, eachconfigured to perform certain functions within the electronic device.

The memory may include one or more internally fixed storage units,removable storage units, and/or remotely accessible storage units. Thevarious storage units may include any combination of volatile memory andnon-volatile memory. The storage units may be configured to store anycombination of information, data, instructions, software code, etc. Moreparticularly, the storage devices may include haptic effect profiles,instructions for how the haptic actuation devices of the input/outputdevices are to be driven, or other information for generating hapticeffects.

In addition to the touch sensitive screen, the input/output devices mayalso include specific input mechanisms and output mechanisms. Forexample, the input mechanisms may include such devices as keyboards,keypads, cursor control devices (e.g., computer mice), or other dataentry devices. Output mechanisms may include a computer monitor, virtualreality display device, audio output device, printer, or otherperipheral devices. The input/output devices may include mechanisms thatare designed to not only receive input from a user and but also providefeedback to the user, such as many examples of touch screen devices. Thetouch sensitive screen and other input/out devices may include anysuitable combination and configuration of buttons, keypads, cursorcontrol devices, touch screen components, stylus-receptive components,or other data entry components. The touch sensitive screen may alsoinclude any suitable combination of computer monitors, display screens,touch screen displays, haptic or tactile actuators, haptic effectdevices, or other notification devices for providing output to the user.

FIG. 4 is a cut through a part of one exemplary force sensitivetouchscreen 310. The screen 310 may consist of a sandwich of liquidcrystal 312 between a top glass substrate 304 and a bottom glasssubstrate 318 with polarizers 302, 320 on the external surfaces of theglass substrates 304, 318. On the internal surface of the top glasssubstrate 304 is a colour filter 306. A first layer of strips oftransparent electrodes 308 is on the top glass substrate 304. A secondlayer of transparent electrodes 316 is attached on the internal surfaceof the bottom glass substrate 318, perpendicular to the first layer ofelectrodes 308. Therefore if the first layer of electrodes ran in adirection parallel to the width (commons) of the glass substrates 304,318, then the second layer of electrodes 316 runs in a directionparallel to the length (segments) of the glass substrates 304, 318.Wherever a strip from the first layer of electrodes 308 crosses a secondstrip from the second layer of electrodes 316, a pixel element isformed.

The electrical model of a pixel is similar to a capacitor. Theintersection of segments and commons of strip 308, 316 form capacitorplates and the liquid crystal 312 acts as the dielectric of a capacitor(C). Voltage V across this capacitor is equal to charge Q overcapacitance C (V=Q/C), therefore, voltage is proportional to thedistance between the plates. When a force is applied to the surface ofthe top glass, i.e. substrate 304, such as a press with a finger orstylus, the distance between the top and bottom glass substrates 304,318 changes and thus the distance between the strip electrodes 308, 316changes. When the distance between the electrodes changes, thecapacitance of the pixel changes and the change in capacitance can bedetected by the resulting change in voltage at that pixel. Because ofthe relationship between voltage and distance between the plates, as theelectrodes get closer, the pixel voltage will decrease. Using thecapacitance change of a pixel the location and amount of an appliedforce may be measured.

The invention may also be used in devices, which sense the distancebetween the screen surface and the pointing object.

Another embodiment of a portion of a touchscreen 410 is shown in FIG. 5,which illustrates a touchscreen 410 for generating electrical signalsindicating a positional property and an extent property of a mechanicalinteraction within a sensing zone. Touchscreen 410 comprises a pluralityof conductive layers that comprises at least a first conductive layer402 and a second conductive layer 403. At least one of the plurality ofconductive layers is a force sensitive conductive layer, which maycomprise a quantum tunnelling conductance (qtc) material. Thetouchscreen 410 is configured such that contact between conductivelayers is allowed during the absence of a mechanical interaction withinthe sensing zone.

The plurality of conductive layers of the touchscreen is provided withan arrangement of electrical terminals (not shown). The electricalterminals may be arranged to provide the touchscreen with athree-terminal sensing arrangement, to allow a single positional valueand an extent value of a mechanical interaction to be determined. InCartesian co-ordinates, a three-terminal sensing arrangement allows ameasurement in the X-axis or Y-axis direction, along with a measurementin the Z-axis direction. The electrical terminals may be arranged toprovide the touchscreen with a four-terminal sensing arrangement, toallow first and second positional values and an extent value of amechanical interaction to be determined. In Cartesian co-ordinates, afour-terminal sensing arrangement allows a force measurement in theX-axis direction and the Y-axis direction, along with a forcemeasurement in the Z-axis direction.

The touchscreen 410 may be connected to the interface module 404 inelectrical connection with the electrical terminals of the plurality ofconductive layers. Touchscreen 410 is configured to respond to amechanical actuator. In one embodiment, the touchscreen is configured tobe responsive to actuation by a finger 405.

In some embodiments, a touch surface can be provided in areas of theelectronic device outside of the touchscreen. In yet furtherembodiments, the touch surface may be an external touchpad that can beconnected to the cell phone (or other computational device) by means ofa cable to a cable connector or a wireless transceiver.

A three-dimensional sensing in a volume above the display 110 of adevice 100 to detect gesture together with suitable user interface (UI)is illustrated in schematic of FIG. 6. The UI may be 3D(three-dimensional) as well and also be used together with a 3D displayor a projector.

According to this embodiment an object, e.g. a user's finger is sensedin a 3D volume by using capacitive or electric field sensing. FIG. 6illustrates a device 550 for capacitive and electric field sensing basedon transmitting a signal 560 by means of one or several electrodes 551and then receiving the response with another electrode(s) 552. Theelectrodes may be arranged behind a display layer 553 and controlled bya controller 554. If an object is close enough to the touch surface, achange in the capacitive coupling between the electrodes and the groundwill be detected as the received signal strength will change.

In 3D detection embodiments, the distance change between the electrodeand the pointing object is interpreted as pressure change. The pressurechange affects the haptic feedback as described earlier.

As used herein, a “touch-screen” refers to a screen or portion of ascreen which may be part of a touch input device with an associatedimage display having a “touch surface”. A touch surface may be integralparts of an electronic device, such as a touch screen display, or aseparate module which can be coupled to the electronic device by a wiredor wireless data link.

The various embodiments of the present invention described herein isdescribed in the general context of method steps or processes, which maybe implemented in one embodiment by a computer program product, embodiedin a computer-readable medium, including computer-executableinstructions, such as program code, executed by computers in networkedenvironments. A computer-readable medium may include removable andnon-removable storage devices including, but not limited to, Read OnlyMemory (ROM), Random Access Memory (RAM), compact discs (CDs), digitalversatile discs (DVD), etc. Generally, program modules may includeroutines, programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data types.Computer-executable instructions, associated data structures, andprogram modules represent examples of program code for executing stepsof the methods disclosed herein. The particular sequence of suchexecutable instructions or associated data structures representsexamples of corresponding acts for implementing the functions describedin such steps or processes.

Software and web implementations of various embodiments of the presentinvention can be accomplished with standard programming techniques withrule-based logic and other logic to accomplish various databasesearching steps or processes, correlation steps or processes, comparisonsteps or processes and decision steps or processes. It should be notedthat the words “component” and “module,” as used herein and in thefollowing claims, is intended to encompass implementations using one ormore lines of software code, and/or hardware implementations, and/orequipment for receiving manual inputs.

It should be noted that the word “comprising” does not exclude thepresence of other elements or steps than those listed and the words “a”or “an” preceding an element do not exclude the presence of a pluralityof such elements. It should further be noted that any reference signs donot limit the scope of the claims, that the invention may be implementedat least in part by means of both hardware and software, and thatseveral “means”, “units” or “devices” may be represented by the sameitem of hardware.

The foregoing description of embodiments of the present invention, havebeen presented for purposes of illustration and description. Theforegoing description is not intended to be exhaustive or to limitembodiments of the present invention to the precise form disclosed, andmodifications and variations are possible in light of the aboveteachings or may be acquired from practice of various embodiments of thepresent invention. The embodiments discussed herein were chosen anddescribed in order to explain the principles and the nature of variousembodiments of the present invention and its practical application toenable one skilled in the art to utilize the present invention invarious embodiments and with various modifications as are suited to theparticular use contemplated. The features of the embodiments describedherein may be combined in all possible combinations of methods,apparatus, modules, systems, and computer program products.

1. An electric device comprising: a force and touch sensitive screenconfigured to detect and output a detection signal representing a forcegenerated by an object interacting with the force and touch sensitivescreen; a haptic feedback element configured to generate a hapticfeedback; a controller configured to control the haptic feedback elementbased on the force generated by the object interacting with the forceand touch sensitive screen.
 2. The device of claim 1, further comprisinga first filter element arranged between the force and touch sensitivescreen and the controller, wherein the first filter element isconfigured to filter out the force generated by the object interactingwith the force and touch sensitive screen from a force generated by thehaptic feedback.
 3. The device of claim 1, further comprising a signalinput to the controller from the feedback element.
 4. The device ofclaim 1, wherein the first filter element is a low pass filter.
 5. Thedevice according to claim 1, wherein the force and touch sensitivescreen comprises means for generating an electrical signal indicating apositional property of the object and an extent property of a mechanicalinteraction within a sensing zone.
 6. The device according to claim 1,further comprising a sensor for capacitive and electric field sensingbased on transmitting a signal by means of one or several electrodes andreceiving a response with another electrode(s).
 7. The device accordingto claim 1, wherein said controller is configured to define the force ona surface of the screen as a distributed force on an area and therebydetermining relation of the screen surface against a surface andadapting the haptic feedback.
 8. The device according to claim 1,wherein the feedback further comprises audible and/or visual feedback.9. The device according to claim 1, wherein the electric device is amultimedia device for displaying images.
 10. The electric device ofclaim 1, wherein the controller is configured to control the amplitudeof the haptic feedback based on the force generated by the objectinteracting with the force and touch sensitive screen.
 11. The electricdevice of claim 1, wherein the controller is configured to control thefrequency of the haptic feedback based on the force generated by theobject interacting with the force and touch sensitive screen.
 12. Theelectric device of claim 1, further comprising a second filter elementconfigured to receive a control signal output from the controller,filter the control signal, and output the filtered control signal,wherein the feedback element is configured to receive the filteredcontrol signal output from the second filter element and generate thehaptic feedback based on the filtered control signal.
 13. The device ofclaim 12, wherein said second filter element is a band-pass or high-passfilter.
 14. The electric device of claim 10, wherein the amplitude ofthe haptic feedback is proportional to the force generated by the objectinteracting with the force and touch sensitive screen.
 15. The electricdevice of claim 11, wherein the frequency of the haptic feedback isproportional to the force generated by the object interacting with theforce and touch sensitive screen.
 16. An electric device comprising: aforce and touch sensitive screen configured to detect and output adetection signal representing a force generated by an object interactingwith the force and touch sensitive screen; a haptic feedback elementconfigured to generate a haptic feedback; a controller configured tocontrol the haptic feedback element based on the force generated by theobject interacting with the force and touch sensitive screen, whereinthe amplitude of the haptic feedback is dependent on the force generatedby the object interacting with the force and touch sensitive screen,wherein the detection signal is proportional to a magnitude of the forcegenerated by the object interacting with the force and touch sensitivescreen.
 17. The electric device of claim 16, wherein the amplitude ofthe haptic feedback is proportional to the force generated by the objectinteracting with the force and touch sensitive screen.
 18. An electricdevice comprising: a force and touch sensitive screen configured todetect and output a detection signal representing a force generated byan object interacting with the force and touch sensitive screen; ahaptic feedback element configured to generate a haptic feedback; acontroller configured to control the haptic feedback element based onthe force generated by the object interacting with the force and touchsensitive screen, wherein the frequency of the haptic feedback isdependent on the force generated by the object interacting with theforce and touch sensitive screen; wherein the detection signal isproportional to the force generated by the object interacting with theforce and touch sensitive screen.
 19. The electric device of claim 18,wherein the frequency of the haptic feedback is proportional to theforce generated by the object interacting with the force and touchsensitive screen.
 20. A method for controlling haptic feedback for anelectronic device, the method comprising: a controller for controlling ahaptic feedback element based on a force generated by an objectinteracting with a force and touch sensitive screen, wherein thecontroller controls the haptic feedback element to produce an amplitudeof haptic feedback that is proportional to the force generated by theobject interacting with the force and touch sensitive screen.
 21. Themethod according to claim 20, the method further comprising thecontroller controlling the haptic feedback element in accordance with adetection signal received from the force and touch sensitive screen,wherein the detection signal represents a force generated by an objectinteracting with the force and touch sensitive screen
 22. A method forcontrolling haptic feedback for an electronic device, the methodcomprising: a controller for controlling a haptic feedback element basedon a force generated by an object interacting with a force and touchsensitive screen, wherein the controller controls the haptic feedbackelement to produce a frequency of haptic feedback that is proportionalto the force generated by the object interacting with the force andtouch sensitive screen.
 23. The method according to claim 22, the methodfurther comprising the controller controlling the haptic feedbackelement in accordance with a detection signal received from the forceand touch sensitive screen, wherein the detection signal represents aforce generated by an object interacting with the force and touchsensitive screen